1 Bayesian Regularization

Recap the Bayesian regression in the model: $$y=X\beta +\epsilon ,{\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2),{\beta }_j\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathrm{Unif}(-C,C)$$ for a large ${\displaystyle C}$. We have showed in here that $$\begin{array}{}\text{(4.1)}& \beta |\mathrm{data},\sigma \sim \mathcal{N}((X^{\mathrm{T}}X{)}^{-1}{X}^{\mathrm{T}}y,{\sigma }^2(X^{\mathrm{T}}X{)}^{-1})\end{array}$$ when ${\displaystyle C\to \mathrm{\infty }}$.

A different prior is ${\displaystyle {\beta }_j\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,C)}$, then $$\begin{array}{}\text{(4.2)}& \beta |\mathrm{data},\sigma \sim \mathcal{N}({(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+\frac{I}{C})}^{-1}\frac{X^{\mathrm{T}}y}{{\sigma }^2},{(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+\frac{I}{C})}^{-1}).\end{array}$$
When ${\displaystyle C\to \mathrm{\infty }}$, (4.2) is (4.1).

Now consider prior ${\displaystyle {\beta }_0,{\beta }_1\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,C),{\beta }_2,\cdots ,{\beta }_{n-1}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\tau }^2)}$ for a small ${\displaystyle \tau }$, i.e. ${\displaystyle \beta \sim \mathcal{N}(0,Q),Q=\mathrm{diag}(C,C,{\tau }^2,\cdots ,{\tau }^2)}$, and $$\begin{array}{}\text{(4.3)}& \beta |\mathrm{data},\sigma \sim \mathcal{N}({(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}\frac{X^{\mathrm{T}}y}{{\sigma }^2},{(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}).\end{array}$$

(4.2) and (4.3) will be proved later.

The posterior mean is then $$\begin{array}{}\text{(4.4)}& {(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}\frac{X^{\mathrm{T}}y}{{\sigma }^2}=(X^{\mathrm{T}}X+{\sigma }^2{Q}^{-1}{)}^{-1}{X}^{\mathrm{T}}y.\end{array}$$ (Note this is closely related to (2.1)) Note that ${\displaystyle Q^{-1}=\mathrm{diag}\{C^{-1},C^{-1},{\tau }^{-2},\cdots ,{\tau }^{-2}\}}$. When ${\displaystyle C}$ is large, ${\displaystyle Q^{-1}\approx \frac{1}{{\tau }^2}J}$, so (4.4) becomes ${\displaystyle {(X^{\mathrm{T}}X+\frac{{\sigma }^2}{{\tau }^2}J)}^{-1}{X}^{\mathrm{T}}y}$, which matches 2.1 if ${\displaystyle \lambda =\frac{{\sigma }^2}{{\tau }^2}}$.

2 Bayesian Approach for Dealing with Unknown ${\displaystyle {\displaystyle \tau }}$ and ${\displaystyle {\displaystyle \sigma }}$

Assume ${\displaystyle \log \tau ,\log \sigma \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathrm{Unif}(-C,C)}$, ${\displaystyle \beta |\tau ,\sigma \sim \mathcal{N}(0,Q)}$. ${\displaystyle Q}$ is same as above. Then prior joint density is $$\begin{array}{rl}{f}_{\beta ,\tau ,\sigma }(\beta ,\tau ,\sigma )& =f_{\tau }(\tau )f_{\sigma }(\sigma )f_{\beta |\tau }(\beta )\\ & =\frac{I\{{\mathrm{e}}^{-C}<\tau ,\sigma <{\mathrm{e}}^C\}}{4C^2\tau \sigma }{\left(\frac{1}{\sqrt{2\pi }}\right)}^n\frac{1}{\sqrt{\det Q}}\exp (-\frac{1}{2}{\beta }^{\mathrm{T}}{Q}^{-1}\beta )\\ & \propto \frac{1}{\tau \sigma }\frac{1}{\sqrt{\det (Q)}}\exp (-\frac{1}{2}{\beta }^{\mathrm{T}}{Q}^{-1}\beta ).\end{array}$$
(Indicator has also been ignored because ${\displaystyle C}$ is large). The likelihood is $${\left(\frac{1}{\sqrt{2\pi }}\right)}^n{\sigma }^{-n}\exp (-\frac{1}{2{\sigma }^2}||y-X\beta |{|}^2),$$ then the posterior is $$f_{\beta ,\tau ,\sigma |\mathrm{data}}(\beta ,\tau ,\sigma )\propto \frac{{\sigma }^{-n-1}{\tau }^{-1}}{\sqrt{\det (Q)}}\exp (-\frac{1}{2}(\frac{1}{{\sigma }^2}||y-X\beta |{|}^2+{\beta }^{\mathrm{T}}{Q}^{-1}\beta )).$$
Convert the power to quadratic: $$\begin{array}{rl}& \frac{1}{{\sigma }^2}||y-X\beta |{|}^2+{\beta }^{\mathrm{T}}{Q}^{-1}\beta \\ =& \frac{y^{\mathrm{T}}y}{{\sigma }^2}-\frac{2{\beta }^{\mathrm{T}}{X}^{\mathrm{T}}y}{{\sigma }^2}+{\beta }^{\mathrm{T}}(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})\beta \\ =& (\beta -\mu {)}^{\mathrm{T}}(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})(\beta -\mu )+\frac{y^{\mathrm{T}}y}{{\sigma }^2}-{\mu }^{\mathrm{T}}(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})\mu \\ =& (\beta -\mu {)}^{\mathrm{T}}(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})(\beta -\mu )+\frac{y^{\mathrm{T}}y}{{\sigma }^2}-\frac{y^{\mathrm{T}}X}{{\sigma }^2}{(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}\frac{X^{\mathrm{T}}y}{{\sigma }^2},\end{array}$$ where $$\mu ={(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}\frac{X^{\mathrm{T}}y}{{\sigma }^2}.$$ Plug into posterior: $$\begin{array}{rl}{f}_{\beta ,\tau ,\sigma |\mathrm{data}}(\beta ,\tau ,\sigma )\propto & \frac{{\sigma }^{-n-1}{\tau }^{-1}}{\sqrt{\det (Q)}}\exp (-\frac{1}{2}(\beta -\mu {)}^{\mathrm{T}}(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})(\beta -\mu ))\\ & \cdot \exp (-\frac{y^{\mathrm{T}}y}{2{\sigma }^2})\exp \left(\frac{y^{\mathrm{T}}X}{2{\sigma }^2}{(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}\frac{X^{\mathrm{T}}y}{{\sigma }^2}\right).\end{array}$$
The dependence on ${\displaystyle \beta }$ is simple through the quadratic which implies $$\beta |\mathrm{data},\sigma ,\tau \sim \mathcal{N}(\mu ,{(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}).$$ This proves (4.2) and (4.3). Also $$f_{\tau ,\sigma |\mathrm{data}}(\tau ,\sigma )\propto \frac{{\sigma }^{-n-1}{\tau }^{-1}}{\sqrt{\det Q}}\sqrt{det{(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})}^{-1}}\exp (-\frac{y^{\mathrm{T}}y}{2{\sigma }^2})\exp \left(\frac{y^{\mathrm{T}}X}{2{\sigma }^2}(\frac{X^{\mathrm{T}}X}{{\sigma }^2}+Q^{-1})\frac{X^{\mathrm{T}}y}{{\sigma }^2}\right).$$

3 Comments on Bayesian Regularization

In practice, ${\displaystyle f_{\tau ,\sigma |\mathrm{data}}}$ tends to prefer ${\displaystyle \tau }$ neither too small nor too large. Because $$f_{\tau ,\sigma |\mathrm{data}}(\tau ,\sigma )\propto f_{\mathrm{data}|\tau ,\sigma }(\tau ,\sigma )f_{\tau ,\sigma }(\tau ,\sigma ),$$ and ${\displaystyle f_{\tau ,\sigma }(\tau ,\sigma )}$ is quite flat. (Note that there is a big difference between ${\displaystyle f_{\mathrm{data}|\beta ,\sigma }(\mathrm{data})}$ and ${\displaystyle f_{\mathrm{data}|\tau ,\sigma }(\mathrm{data})}$)
Maximizing ${\displaystyle f_{\mathrm{data}|\beta ,\sigma }(\mathrm{data})}$ leads to the unregularized LS estimate leading to overfitting. On the other hand, maximizing ${\displaystyle f_{\mathrm{data}|\tau ,\sigma }(\mathrm{data})}$ leads to a fairly small estimate of ${\displaystyle \hat{\tau }}$ leading to a smooth trend function. This can be understood by noticing $$f_{\mathrm{data}|\tau ,\sigma }(\mathrm{data})=\int f_{\mathrm{data}|\beta ,\sigma }(\mathrm{data})f_{\beta |\tau }(\beta )\mathrm{d}\beta .$$ When ${\displaystyle \tau }$ is large, ${\displaystyle f_{\beta |\tau }(\beta )}$ will be small simply because the normal density with ${\displaystyle {\tau }^2}$ will be flat for large ${\displaystyle \tau }$. When ${\displaystyle \tau }$ is too small, ${\displaystyle f_{\beta |\tau }(\beta )}$ will be significant only for very smooth ${\displaystyle \beta }$ but these ${\displaystyle \beta }$ will have poor values for ${\displaystyle f_{\mathrm{data}|\beta ,\sigma }(\mathrm{data})}$.

Model ${\displaystyle y={\beta }_0+{\beta }_{1}x}$. But how to estimate ${\displaystyle {\beta }_0,{\beta }_1}$?
Well, given a guess, I do know how "bad" it is.

Denote our footprint lengths as ${\displaystyle x_1,\cdots ,x_n}$, and heights as ${\displaystyle y_1,\cdots ,y_n}$.
If ${\displaystyle {\beta }_0,{\beta }_1}$ are known, we predict heights as ${\displaystyle {\hat{y}}_1,\cdots ,{\hat{y}}_n}$, with ${\displaystyle {\hat{y}}_i={\beta }_0+{\beta }_1{x}_i}$.

Define Loss Function: ${\displaystyle L({\beta }_0,{\beta }_1)=\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2=\sum _{i=1}^n[y_i-({\beta }_0+{\beta }_1{x}_i){]}^2.}$

Our ultimate goal is to minimize loss function. $$\begin{array}{}\text{(1)}& & \{\begin{array}{rl}& \frac{\partial L}{\partial {\beta }_0}({\beta }_0,{\beta }_1)=0,\\ & \frac{\partial L}{\partial {\beta }_1}({\beta }_0,{\beta }_1)=0.\end{array}\end{array}$$

Denote ${\displaystyle \bar{x}=\frac{1}{n}\sum _{i=1}^n{x}_i,\bar{y}=\frac{1}{n}\sum _{i=1}^n{y}_i,\overline{xy}=\frac{1}{n}\sum _{i=1}^n{x}_i{y}_i,\overline{x^2}=\frac{1}{n}\sum _{i=1}^n{x}_i^2}$, then the solution is $$\begin{array}{}\text{(2)}& {\beta }_1=\frac{\bar{x}\cdot \bar{y}-\overline{xy}}{(\bar{x}{)}^2-\overline{x^2}},{\beta }_0=\bar{y}-\bar{x}{\beta }_1.\end{array}$$